Improving your max-diff data

Editor's note: Morris Wilburn is a marketing research consultant at Advanced Customer Analytics, Lawrenceville, Ga.

Most studies using maximum difference scaling (max-diff) contain too many questions for any one respondent to answer. This is often dealt with by randomly drawing a subset of questions for each respondent. At first thought, this seems like a reasonable solution; in many other areas of research we select or assign things randomly to avoid bias. But it is not appropriate here. It yields interviews in which, at the respondent level, some attributes are seen much more frequently than others, and some are not seen at all. This occurs even when the attributes are perfectly balanced at the total sample level. This negatively affects the outcome of the analysis.

Consider the following. A typical design for a max-diff study is found in Exhibit 1. Using classical experimental design terminology, this is an incomplete block design. Each row is a question. The columns contain codes representing the attributes; there are 21 attributes, coded “1” to “21.” Each question contains five attributes (alternatives).

Incomplete block designs are constructed to be as balanced as possible. By definition, a design is perfectly balanced if it has two characteristics:

• every attribute appears an equal number of times across questions; and
• every attribute appears with each other attribute an equal number of times across questions.

Exhibit 1 – Typical Max-Diff Design 

The number of times each attribute appears across the 45 questions is shown in Exhibit 2.

Exhibit 2 – Frequency Distribution of Attributes Across Questions

Let’s look at an example. The attribute coded “1” appears 11 times across the 45 questions; the attribute coded “4” appears 10 times across the 45 questions.

This design is not perfectly balanced but it is close. (For technical readers, the highest correlation between attributes is .08.)

Forty-five questions are far too many for any one respondent to answer. A common solution is to take the design shown above and program the interview to randomly draw a subset of questions for each respondent. To examine the appropriateness of this practice, I have drawn 10 independent random samples of 10 questions from this design. They are shown in Exhibit 3. I decided on 10 questions, as opposed to a larger number, because of the increasing use of smartphones to answer questionnaires.

Exhibit 3 – Frequency Distribution of Attributes Within Random Samples of Questions

In the first sample drawn, the attribute coded “1” appears once across the 10 questions; in the second sample drawn, the attribute coded “2” appears twice across the 10 questions.

In all 10 interviews the attributes differ to a substantial degree in the frequency with which they appear. In five of the interviews, one or two attributes are never seen by the respective respondent.

Obviously, the trade-off philosophy of max-diff has been compromised. But what are the practical implications of this? A fundamental consequence is that a max-diff design implemented in this manner is rendered inefficient at the respondent level. That is, the questions collectively do not obtain as much information about preference as they could. This is relevant when the max-diff data are analyzed in a way that uses individual respondents’ data to calculate respondent-level data (e.g., Hierarchical Bayes), which is then used in a segmentation analysis. The outcome of the segmentation will be negatively affected by this inefficiency.

Another issue is that data collected in this manner is biased. There are two aspects of this bias. One is that the mere fact that a given attribute appears more often making it more likely to be chosen as most or least important. Admittedly, the direction of this bias differs from one respondent to another, and so the biases will balance each other out to some degree at the aggregate level. But this bias will still be treated as unexplained variation when the data are analyzed, making the analysis results less precise.

The other aspect is that if several of the attributes collectively appear a relatively large number of times, they function as a standard of comparison in the respondent’s decision of how to answer the questions.

Internally balanced groups

A better method of allocating the 45 questions is to use experimental design techniques to divide the questions into internally balanced groups, and randomly assign one group to each respondent. To illustrate, I have divided the 45 questions into five groups of nine questions. The results are shown in Exhibit 4.  

Exhibit 4 – Frequency Distribution of Attributes Within Internally Balanced Groups of Questions

To avoid position bias, the alternatives within question and the questions should be rotated from one respondent to another in the interview.

It is not mathematically possible to achieve perfect internal balance in this case. But this grouping of questions is much more balanced than that produced by the random selection approach discussed earlier.